Next Generation Consistency Enforcement

When more than one worker can access data simultaneously, they usually break data consistency. To enforce data consistency, a huge performance penalty must be paid. This divides system architecture into two directions. One direction considers data consistency less important than “greater goods”, such as availability or partition tolerance, and so sacrifices it. Dropping data consistency enforcement components from a system usually leads to significant performance improvements. However, data consistency is still indispensable and ubiquitous in the real world applications whether the data systems support it or not. As a result, researchers and practitioners started to consider the other direction: high performance consistency enforcement. Recent representative works include data-flow systems, transaction processing on partitioned database and invariant-based concurrency control, etc. Unfortunately, although these studies have achieved considerable progress, there is still a giant performance gap between them and the systems without data consistency at all. Why is protecting data consistency so hard? Based on previous experience, we believe the “no man left behind” promise is the root of all problems. As in most of the RDBMS textbooks, a typical use of databases is online banking. Data consistency (as a part of ACID) is enforced to prevent money being lost. It is a banker’s public relation nightmare if even just one customer’s account is compromised, so every constraint on every tuple must be respected. Recent protocols build on this same commitment; for example, invariant-based concurrency control protects all invariants that it is capable of. However, we are not in the 20th century any more. People have a huge volume of data now. The ideal story is to protect perfect consistency on big data. As this is not feasible, everyone must choose between perfect consistency and big data, and maintaining “no man left behind”-style consistency becomes questionable. We envision the next generation system that protects data consistency when consistency is as important as scalability and performance. One potential scenario involves analytical jobs. There are more and